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ABSTRACT 

This study examined the effectiveness of a metacourse 
consisting of eight lessons interspersed over a semester-long 
beginning course in BASIC and aimed at providing mental models, 
problem-solving strategies, key concepts, and other heuristic 
structures. The experimental group consisted of 6 teachers who taught 
9 clasSteS of a total of 132 high school students; the control group 
consisted of 9 teachers who taught 13 classes of a total of 239 high 
school stuuents. Data collection included a student questionnaire on 
previous experience with computers, a pre/posttest to assess general 
cognitive skills at the beginning of the term and possible transfer 
effects at the end, classroom observations, an end-of -semester test 
on BASIC, and homework assignments. Analysis focused on teachers' 
fidelity to the metacourse lessons, the impact of instruction on the 
students' mastery of BASIC, and the transfer of cognitive skills from 
programming to other domains. Results showed that teachers were 
faithful to the lessons and found them quite teachable. Experimental 
group students made significantly fewer errors on the test of BASIC 
and did significantly better on all major categories of problems. 
Evidence of transfer that was observed was limited to a particular 
problem similar to the programming tasks. The appendices include 
examples from the metacourse manual, the cognitive skills 
pre/posttest, the BASIC test, the classroom observation sheet, and 
the student questionnaire. (34 references) (Author/MES) 
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An Empirisai Study of a "Metacourse" to 
Enhanca tha Laarnlng of BASIC 



Enhancing instmction in an existing subject matter on a wide scale is a conspicuous and 
rarely met challenge of contemporary education, and for understandable reasons. On the one 
hand, the subject matters bring with them a vmvber of conceptual challenges and problems of 
metacognitlve control that require careful study U) disclose and careful instmctional design to 
remedy. On the other hand, however artfully fashioned better instmction may be, wide-scale 
implementation must mT\ the gauntlet of a number of impeding factors - teacher training, cost of 
materials, institutional inertia, and so on. 

A desirable program of research would address notonly the learning of a subject matter but 
also the practical problems of implementation as part of a unified program of inquiry. During the 
past several years, we have tried to conduct such an investigation into the pedagogy of 
programming. Through cfinical studies and teaching experiments, we have sought to understand 
better the factors that interfere with beginnera' mastery of programming in BASIC and LOGO and 
to devise instnictlonal methods that enhance their learning (see e.g. Peridns, Farady, Hancocic, 
Hobbs. Simmons, Tuck, & Villa, 1986; Peifclns, Hanfxck, Hobbs, Martin, & Simmons, 1986; 
Peridns, Hancock, Hobbs, Martin, & Simmons, 1986; Pertdns, Martin, & Farady, 1986). Regarding 
implementatton, we have tried to couch our instmcttonal methods in a fomtat called a 
"metacourse," designed from the first to address some of the daunting problems of wide-scale 
disseminability (Peridns, Farady, Simmons, &Villa; Pertdns, Schwartz, & Simmons, in press). 

This paper describes a large-scale experiment examining the effectiveness of the 
metv^course we have devetoped in enhancing high school students' learning of BASIC. The 
positive results can be taken as support for the general analysis of the difficulties of programming 
developed in our eariler wori^, and for the viability of our approach to educational change. Before 
detailing the experimemal method and results, we describe brief ly the context established by our 
prior wori(. 

Stiiriflntft' leafning pmhtems in mastflrino proorammlna 

It is plain that computer programming poses special challenges. Like mathematics and 
phystos, programming is a predston-intensive subject matter, requiring metteuious care with 
details. Programming is also problem-solving intensive, student activities focusing almost entirely 
on resolving programming problems. While today this is typical of mathematk^s, physks, and 
certain other science subjects, we note that am. school subject can and pertiaps shoukl be 
treated In a problem-solving intensivr way. Rnally, programming is not just problem-solving 
intensive but "design intensive." That is, students have to constmct whole complex products that 
do certain jobs, not jus! derive particular answers like 35 cm/sec or A«17. In today's schools, 
complex products appear mostly in art (the wori(s themselves), EuclMean geometry (proofs), and 
English and Social Studies (essays). Again, we note that any school subject can and pertiaps 
shoukl emphasize the constroctton of complex products. 
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It Is hardly surprising that a precision-intensive, problem-solving intensive, and design- 
intensive subject matter should give many students considerable difficulty. In recent years, 
evidence has accumulated that a high percentage of students In elementary and high school 
achieve only extremely limited mastery of programming even after a semester or two of instoiction 
(see e.g. Mawby, 1987; Pea, 1986; Pea & Kurland, 1984a; Pea & Kuriand, 1984b; Kuriand, Pea, 
Clement, & Mawby, 1986; Kuriand, Clement, Mawby, & Pea, 1987; Bonar & Soloway, 1985; 
Soloway & Ehriich, 1984; Sleeman, Putnam, Baxter, & Kuspa, 1986). In seeking to summarize 
and synthesize our own and others' findings In this area, we have found it useful to characterize 
students' difficulties under three broad headings: fragile (knowledge, a shortfall in elementary 
problem solving strategies, and problems of confidence and control. 

Frar^te knowifldqa. "Fragile knowledge" refers to the fact that students commonly display 
partial knowledge, conskJerabie inert knowledge (not evoked In contexts of need but retrievable 
with cueing), and gariMed knowledge (concepts used In the wrong place. In Inappropriate hybrids) 
of progratr - j. Pertains, Hancock, Hobbs, Martin, and Simmons (1986) discuss fragile 
programn. y kn^wkxige in detail, emphasizing the importance of distinguishing between fragile 
and missing knowledge: Students typically have much more knowledge than they use well. If 
students couU somehow activate their Inert knowledge and perfonn internal cross-checks of 
garbled knowledge, they might perfomn substantially better. 

Fiflmantary probtem-aoMng strategies . By asking themselves elementary "problem 
managemem* questksns like "What am I trying to do now," "do I know a command that coukl help," 
"exactly what does the line of code I just wrote do if I hand execute it," and so on, students might 
make better use of their fragile knowledge base. Cllnteal experiments reported In Peri<ins, 
Hancock, Hobbs, Martin, and Simmons (1986) and Perkins, Martin, and Farady (1986) suggest 
that this Is so. Unfortunately, students do not appear to probe with such questions as often as 
they might, a shortfall In elementary problem-solving strategies. 

nonfidenRft and ftontmi. Rnally, students Often evince motivational problems that interfere 
with their controlling well their own problenvsolving processes. For example, many students 
simply disengage from programming problems and commence a skle activity or seek help as soon 
as ttie least difficulties emerge. These "stoppers" as we have called them are often quite capable 
of continuing on non-directive prompting, but do not seem to recognize their own abilities 
(Peridns, Hancock, Hobbs, Martin, & Simmons, 1986). 

One way or another, efforts to enhance Instiructton in programming shoukl address the triple 
problem of fragile knowledge, strategk: shortfall, and confklence. Note that these diffteulties as 
described here do not force wholesale reconsUerafion of what is taught In elementary 
programming instnicthsn. After all, ttie key commands, the fundamental operating procedures, 
and so on, surely need to be taught. Instead, the shortfalls klentifled coukl be taken to Invite 
some sort of "booster shof - some treatment to enhance the learning set and strategic repertoire 
students bring to the enterprise of programming. 
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Thft notion nf a mfltacoursft 

With this point in inind, wc introduce the notion of a ''metacourse/ The goal d a metacourse 
is to provide mental models, problem-solving strategies, key concepts, and other structures that 
may help students to understand more deeply and wield more artfully the knowledge they are 
acquiring during their regular instruction in a subject matter In particular, the Metacourse in 
Programming discussed here offers students a mental model of the computer and how it worths at 
a level appropriate for understanding BASIC and its operation, a strategy for understanding new 

^ commands and relating them to the mental model of the computer, several strategies for breaking 

programming problems down into subproblems of varkxjs sorts, and other concepts and tacttos 

V designed to help students deal with the difficutties kientified eariier (Perkins, Schwartz, & 

Simmons, in press). 

The metacourse is not a course - a complete remaking of the usual instructton. Rather it 
functtons in a "meta" way, infusing a few important and often neglected conceptual elements into 
Ixjsiness as usual." Thus, the Programming Metacourse is organized to altow the teacher to 
introduce key concepts periodtoally as the term advances and students gain a knowledge base in 
BASIC. This infuston process, together with the metacognitlve emphasis, are the factors that 
distinguish metacourse design from conventtonal cunrteulum redesign. 

Why might a metacourse offer an approach to enhancing instmction th^ evades some of the 
inplementatton diffk:utties outlined eariier? In comparison with a new course altogether, a 
metacourse is much more connpact and its; materials much less expensive. A metacourse does 
not displace, but merges smoothly with, existing materials and instmctional practtoes. 
Accordingly, we suggest that a metacoun^ lends itself to wkle-scale dissemination easily more 
than most efforts at curriculum refonn. 

The results of a preliminary study reported in Peridns,Farady, Simmons and Villa (1986) 
indteated that the Metacourse was eminently teachable, with no major problems of teacher 
preparatton. WhHe there was some enoDuraging evkJence that the Metacourse was having its 
intended effect, the data from this fdnfnalive study were far from condustve. 

In the fbltowing pages, wc report on a large scale experiment involving a number of treatment 
and control classroom attempts to provide further evklence on the effectiveness of the 
Metacourse in settings where emphasis; is placed on infusing the key concepts into the teacher's 
presentattons throughout the entire course. 



Method 

Setting 

The Programming Group's second empirical study of the Metacourse was conducted within 
the context of another Educattonai Technotogy Center study, the "Laboratory Sites" project. This 
broader study, described in detail in another report, aimed to involve the teachers using the 
Metacourse (as we!! as inten^entions in two other subject matter areas) in the discusston and 
evaluation of the educattonal innovattons they were implementing in their classrooms. 
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Participation in the Laboratory Sites project meant that the teachers using the Metacourse 
received mors support in their efforts to adapt the new materiais to their own styles and 
classrooms than would normally t)e the case. The main extra supports were: monthly meetings 
with the research team to discuss issues pertaining to the implementation of the Metacourse; 
frequent contact with a research assistant or the Laboratory Sites project liaison assigned to the 
Programming Group; access to all Educationai Technology Center personnel, as well as each 
other, via a personal computer communications network and a small stipend in compensation for 
their participation in collaborative research. 

Although not all of these supports may have been significant for all of the teachers in the 
experimental group, fallen together they constitute an enriched support environment not 
nomnally present, and not present in our control sites. The control site teachers, however, were 
not faced with the problem of introducing and infusing new. innovative materials into their wel! 
practiced normal BASIC curriculum. 

From the standpoint of the research, the participation of the Programming Group in the 
Laboratory Sites project created opportunities for very precise and timely feedbadc from teachers 
about the effectiveness of the different lessons in the Metacourse. This feedback, which would 
not othenvise have been available proved to be extremely valuable in gukjing further revisions to 
the Metacourse. 

Siihjects 

The experimental group consisted of 6 teachers of BASIC, who taught 9 classes and 132 
students at 5 high school laboratory sites. The control group pool consisted of 9 teachers who 
taught 13 classes and 239 students at 8 control sites. The large control group was used so that 
the groups coukl be matched if necessary on the basis of the general cognitive prs-test if overall 
pre-test scores indicated that one group or the other was significantiy more able. The difficulties 
in data analysis encountered in the first empirical study, in which the control group proved much 
more able on the general cognitive measures, might thus be avoUed. 

All teachers were experienced programming instnjctors. beginning the semester with at least 
2 years of previous experience teaching BASIC classes. All of the classes were straightforward 
programming classes, meeting on average 5 times a week for forty-minute periods throughout 
one semester. 

Pronedtires 

Teach'mg Interventbn 

All of the experimental group instmctors met with the Programming Group staff and research 
assistants earty in the semester in order to go over the Metacourse materials and procedures. 
Each teacher was introduced to a particular research assistant who wouid observe two 
Metacourse lessons and one non-Metacourse lesson. 
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As described in the introduction, tlie Metacourse is a series of instmctional lessons 
designed to enhance and suppiement the material typically covered In a first-semester BASIC 
course. Cne major concern is to provide students with a clear rmntal trxxlei of the computer in 
the BASIC environment and to help students internalize and employ the model when appropriate. 
Another goal of the Metacourse is to provide students with programming-spedfic si<ills and a 
conceptual framewori( that guide the student in initial understanding and subsequem application 
of material learned in class. 

The eight lessons are designed to be interspersed throughout the semester. Preliminary 
lessons provide students with a visual model of the computer and equip students with the 
V conceptual f rameworit for understanding commands, a framewori( that involves thinidng of a 

statement in terms of its purpose, syntax, and action (semantics). Later lessons deal with 
strategies for writing good programs including breaking programs down into manageable chuni<s, 
thinking of code in temra of functional units called "l^attems", and regular use of checking and 
debugging strategies. The components of the Metacourse are described in more detail betow. 

Thft -paper eom piitftfr A vteiial modftl of th» comntiter. The concept of a mental model is 
key in the design of the Metacourse. Recent wortt in the f iekJ of cognitton underscores the 
importance of helping students construct robust models in various domains (cf. Beveridge & 
Paridns, 1987; Gentner & Stevens, 1963; Johnson- Laird, 1983; Mayer, 1976, 1981). Further, 
empirical woric in the pedagogy of computer programming suggests tTiat a stronger mental model 
of the computer can result in Increased programming perfonnance (DuBoulay, 1986; OuBoulay, 
O'Shea. & Monk, 1981 ; Mayer, 1976, 1981.1985). 

In the Metacourse, students learn a visual model of what happens inskie the computer 
during program executton to help them interpret exactly what the various BASIC commands do 
(See Appendix A). The model is designed to help counter some of the problems of fragile 
knowiedge and strategy shortfall we noted In students in our clinical woric Regular use of the 
model iH learning commands and in envisioning the effects of single lines or chunks of code 
shouki sen^e to help buikJ a robust knowledge base of command effects. In addition, the modal 
can provMe students with a f airiy simple strategic tool for debugging. Using the visual model to 
imagine precisely the actions of the computer at a trouble spot gives stoppers a tool to help them 
get moving again and may assist haphazard movers by provUing a focus for their activities. 

Speciftoaliy, the model depots variables and their vakjes, charactere on the screen, and f tow 
of control. The model is functional rather than technical in nature in an effort to promote visibility 
and simpltolty in the model (cf. DuBoulay, O'Shea, & Monk, 1981 ; Beveridge & Parkins, 1987). 
Thus, for example, the student thinks of a variable as the name of a box in memory with a number 
or character string in it. This model is called the *paper computer," because students receive 
paper fbmns displaying the visual model on whtoh they hand execute programs. In addition, an on- 
line verston of the paper computer is being designed, a tutorial that presents the same visual 
model of the computer and steps through a program line by line, making the recpjisite changes In 
the computer state as each line is executed. 

purpose-syntay-actten: A framewortc for program constituents . In the Metacourse students 
also ieam an analytical scheme for understanding commands and command lines. The scheme 
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encourages students to consider the command's purpose, its legal syntax, and its action in the 
computer wofid as shown on the paper computer. The purpose-syntax-action framework is used 
to help students recognize, attend to, and organize the important features of a new command 
when it is first encountered. Addilionatiy, the framework serves as a problem-solving aid when 
students write programs. In considering how to accompiioii a particular programming task 
students are a!tked to think carsfuRy alaout what commards they know whose purposes may 
accomplish the task at hand. When writing command lines, students are encouraged to attend to 
the syritax, and to envision the action of the command, that is, the effect of the line on the 
computer. Students are also encouraged to use a "minknanual", a quick reference gukle that 
includes the key BASIC conmiands organized according to the purpose, syntax, and acthn 
c|uestk>n8, with examples (See Appendix B). As students gain programming experience and 
t>ecome oomfortat)le in usirig the framework for individual commands, they are introduced to the 
utility of emptoying a similar organizatkxiai strategy for iarger "patterns" of code. (Patterns will be 
discussed betow.) 

The choice of temns in the framework is derived in part from the contrast between 
programming pragmatk», syntax, and semantics, and in part from a model for leaning devek^ped 
by Peridns (Psrkins, I986a,b). Purpose has a major role in the triad as a means to counter some of 
the probiwms f rsgle knowledge, spedRcaliy that of Inert knowledge. In eartier dinfeal studies it 
was found that often students had a knowledge of relevant command stnjctures, but coukj not 
retrieve them, apparently ftiing to nMke the connectton between what needed to happen in the 
progrsim and the commands that wouM serve the purpose (Peridns & Martin, 1986; Peridns, 
Martin & Farady. 1986). The attentkm to syntax is based on eariier observattons noting the degree 
to whtoh BASIC and LOGO program perfonnance among novk» programmers can be affected by 
problems of a purely syntactto nature (Peridns, Farady, Hancock, IHobbs, Simmons, Tuck, & Villa, 
1986; Perkins, Farady, & Martin, 1986). The emphasis on the action or semantk» of a command 
t\m stems from the desire to help students constnjct and utilize a robust knowledge base of 
command effects as described above in the discussk^n of visuaS models. 

Pattiima; thinking bayond indivklual commanda. The Metacourse emphasizes the 
importance of organizing programming knowiedgo not only in tenns of indivkluai commands but 
also in terrns of nfuldpie Hnes of code that wori( together to accomplish a partk:ular task or s^^ 
Such recurrent schema are caHed "patterns." a term roughly synonomous with the iprogramming 
plans" described by Sotoway and colleagues (Sotoway & Ehrik:h, 1984: Joni & Sotoway, 1986). 
Patterns provkJe an bitennediate level of analysis between the whole program and individual 
command lines, and offer a way of helping students to organize and coniprehend code used for 
frequent programming tasks such as counter variables, certain compound condittonal branches, 
and so forth (See Appendix C). The Metacourse stresses the importance of patterns as an 
intennediate level coding strategy as well as a debugging strategy. 

Writing a program: Wanning, coding, chacklnty. and dftbiigg ln q . Test results and teacher 
comments from our prevkHiS study indk^ed that students needed addi^nal instnjctton in 
planning and writing whole programs. TYhjs the revised Metacourse provkies students with a 
procedure for writing progrann that emphasizes the importance of planning, checking, and 
debugging in addHton to the actual coding of a program. Another major point of emphasis is that 
writing programs is a process of refinement, involving multiple rounds of these four activities. 
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As a first step in top-level planning, students are encoursged to think In tenns of 
"interactions' with the user. The interaction refers to the sequence of coirputer outputs and user 
inputs that occurs as a user uses a ptoQram. This heuristic offers a concrete laeginning point for 
the novice programmer, who frequently experiences diffiumies in rnoving from a given problem 
statement to the initial stages of the task decomposition. For many pet irams, an outline 
accorcSng to the 'rounds' of interaction provkies a decomposition into sutxjnits that amount to 
subprobiems in the programming task. An initial focus on interacttons as the student begins to 
procram can help to avoM the frequent problem of moving from a problem statement directly to a 
coding phase without sufficient thought and effort devoted to planning. 

« From this inttial breakdown of the problem, students are asked to conceptualize program 

sections in terms of the patterns that might serve the sectton purposes. The patterns provide the 
student with the tools to manage and create code above the level of asingle command line. 
StudentsaretaughttheutlBtyof having a repertoke of patterns that arspoitabfe across programs. 
Thus, for example. V a stixiem recognizes the need ki a program to implement code that win trap 
unreasonabio inputs from the user, the student can call on a txiHetproofing' pattern to help with 
thattask. This simple pattern inckjdes a condltkNiai branch that rejects inappropriate inputs, 
prompting the use for another input. 

At eadi level of the planning and codbig stages students are encouraged to consider the 
purpose of chunks of code and the action those chunks actually effect Students are taught to 
check their code, mentally simulating the actton of the program line by line to catch the 'easy" 
bugs before the program is actually run and tested. The Metacourse also emphasizes that bugs 
and debugging are an inevitable and integral aspect of program production, not a reflection of 
poor programming. 

Assessment procedures 

A pretest in general cognitive skills was administered at the beginning of the term. At the 
end of the tenn, the cofpiitive skUs test was readministered to test for possble transfer effects, 
and a general BASIC skiSs test was given to test for mastery of BASIC. Both tests are described 
betow. 

noqnltiva Skflte Pretesf.Pnsttftst ^Appanriht T)). A cognitive skills tf^st was devetoped wHh 
two purposes in mind. First, such a measure might be expected to correlate with programming 
perfonnance (most cognitive skills are interrelated), and thus serve as an indfeator of level of 
general student ability k\ comparing treatment and control students. In additton, it seemed 
sensible to probe the p.3sst>«iity of transfer of cognitive skills. In general, findings on transfer from 
programming have been negative (Kurtand, Pea. Clement. & Mav.ty. 1986; Mawby. in press: Pea 
& Kuriand, 1984a,b: Soksmon & Perkins, in p?sss), nor was the Metacourse specif k:ally designed 
to pmmoXo transfer, Kcwever. there have been occastonal positive findings (Clements. 
Cli- * > Its a CuVo, 1984) and the issue has great currency, warranting an effort to examine the 
Zc it transfer in tho present study. 

.!n!tive skills instalment wa& designed to test skill in formal syllogistic reasoning, 
i.^ ,^ .& \t leasoning (e.g. If the day after tomorrow were Thursday, what wouW the day 
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before yesterday be?!, field-independence and planning (a task of counting the number of 
triar>ytes in a complex diagram). In addition, tiie test incorporated an algebraic version of the well- 
known students-and-ptofessors problem (Clement, Lochhead, & Monk, 1981) where students 
typkaliy have great diffkxiKy translating a simple algebra word problem into an aigebrak: equatton. 
Soioway and his colleagues have suggested that computer programming experience may help 
students deal with this sort of problem more effectively (Ehrlteh, Sotoway, & Abbot, 1982; 
Sok>way, Lochhead, & Clement, 1982). A further item relaUng to accuracy and precision of 
obsen/atton and descriptton, required the student to accurately describe a cor >lex geometric 
figure in order fOr another student to be able to recognize it anwngst a set of similar complex 
geoiTietric figures. 

The cognitive skills test also included a problem of a type quite similar in character to a 
programming problem involving combining patterns in a program-like way. Students had to 
generate a descriptton of some events using a restricted language containing the words "repeat" 
and "decMe". (The design of these problems was suggested by Ellen Mandinach of the 
Educattonal Testing Servk:e); in prior research, when the rare case of transfer from programming 
has been found, it has emerged most often on tasks with mariced similarity to programming (cf. 
Kuriand, Pea, Clement, & Mawby, 1986; Unn, 1985). 

Rnally, in additton to the purely "cognitive" aspects of the pretest, we also included a series 
of questtons designed to address a paitkxilar attitudinal factor, tocus of control (Rotter, 1966). 
Research in this area (Dwock & Bempechat, 1983; Dweck & Ucht. 1980), as well as studies of 
perfonnance in the programming domain (Perkins, Martin & Farady, 1986; Zelman, 1985), point to 
the importance of attending to feelings of confklence and control in studen! perfomiance. Thus 
we inckjded a 10 questton subtest derived from the "attributton of intellectual responsibility scale" 
(Rotter, 1966), measuring intemality/externality of control in positive and negative intellectual 
situations. 

BASIC Test /Appendix E ) . The BASIC test was a falrty standard paper and pencil 
programming test, comprising 1 6 items. The problems were fonnulated to evaluate certain 
general programing skiUs - the ability to hand-execute, debug, break a problem into subtasks - as 
well as to test knowledge about programming commands typk^ally presented in an introductory 
BASIC course (e.g. PRm, LET, INPUT, FOR/NEXT, IF-THEN). The test was also designed to 
include certain language independent bugs klentified by Pea (Pea, 1986) . 

The first nine problems tested simple productton and recognitton skills, usually requiring 
productton of only one line of BASIC code using one command statement. The last seven 
problems included three hand-executton problems, one debugging problem and three tonger 
programming problems requiring somewhat rnore complex stmctures such setting up counters, 
use of for-next loops and the like. The 16 problems are briefly described betow. 




Simplft one line command problems (1 -9): 

1 . PRINT-string: The student Is asked to write the code that wouki result in a specif to 
string 

being printed on the screen. 

2. PRINT-numben A spedfte number is to be printed on the screen. 

13 
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3. INPUT-number A user is prompted to type in a numtMr. 

4. LET-numberSetavariabletoanumt)er. 

5. LET-expression: Set a variable to the result of an operation. 

6. LET-string: Set a variable to a string. 

7. IF/THEN: If a given condition holds, print an expression. 

8. FOR-loop: fill In blanks to coniplete a simple For/next loop. 

9. GOTO: Rll in a blank to complete a Goto statement. 

Hand execution pmbtems (10. 11 and 13): 

1 0. Given a short program, show what will appear on the screen when the program is 
executed. 

1 1 . Garden path: Same as 10 above, except that what will appear on the screen when 
the program is executed is not what one wouM expect - the hand-executor is led 
down a garden path. 

1 3. Paraileiism bug: Same as 1 0 above, except that the student is given a program in 
which the indivkiual commands are correct, but are presented in the wrong order. 
This was designed to test for skill in recognizing what Pea (1 985) calls a parallelism 
bug. the assumptton by programmers that the computer is able to execute more than 
one line of code simultaneously (in whk:h case order woukl not be important). 

Dflbm^ ini y pmbtem M2 - see also problems 11 and 13): 

1 2. Egocentrism bug: The studer^ is given a program with an INPUT statement missing 
because they often assume that the computer is aware of what they are thinkirig 

Pmdtifttinn probtems (14 - 16): 

1 4. FOR-kx)p: Using a FOR/NEXT toop, write a program that wiH print the numbers 1 
through 10 on the screen. 

15. IF-k)op: Same as problem 14. except that the kx>p is created using an IF statement. 

1 6. Complex: The student is asked to write a more complex program that requires an I 
INPUT statement inside a FOR toop that accumulates a sum. 



Classroom Observations 

Researchers conducted classroom obsen/attons at all treatment s'rtes. systematically 
recording a number of featwes of the classroom dynamics and instructtonai style, as well as items 
related to the Metaoourse treatment itself. Obsen/attons included two Metacourse lessons, as well 
as one regular class, in each case conducted by a single research assistant assigned to observe 
one or two teachers. Assignments of multiple obsen/ers to a class were not made for two reasons. 
Rrst. we wished to be as non-obstmsive as possible and thus each teacher and class coukl get 
comfortable with a single outskJe obsen/er. Second, since our main comparsons were between 
Metacourse and non-Metaccurse lessons in the same dasses. inter-rater reliability was not an 
issue. 

The majority of the items on the obsen/atton instmment (see Appendix F) were rated on 
7-point Ukert scales (where 7 is the highest rating and most desirable score). These included 
situattonal factors such as student- teacher and student-student interactton. student 
responsiveness and engagement, and teacher presentattonal style. Items relating specifically to 
the lUletacourse included fklelity to the Metacourse and infusion of Metacourse concepts into the 
Metacourse lessons. In the area of classroom interacttons. the scales measured amount and 
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quality of interactions t^atween students and teachers and among students* Two types of 
student participation were scored, the degree to which students engaged in process- and 
product*oriented partic^ion. Students (as a group) were also rated on preparedness for the 
material presented. Teachers were also rated on mode of instruction (e.g.. presentational, 
interactive) and on degree of comfort with the material presented. Observers noted whether the 
instruction included any bridging activities between programming and other academic disciplines 
or real-life situations. Rnally, observers used the 7-point scale to give an overall rating of 
effectiveness of instmction. 

In the obsen^ations of Metaoourse lessons, fidelity to the lessons was rated, based on 
adherence to various aspect*^ jf the lesson (introduction, lecture, partidpation, explanation, and 
exerdses). The Metac^urse lessons were also rated according to the degree to which the basic 
principles of the Metaoourse had been integrated into the lesson. 

Student Questionnaire 

At the beginning of the semester, students filled out a short questionnaire designed to help 
assess their previous experience with computers and computer programming (See Appendix G). 
Items were designed to determine the students* general experience with computers in the 
schools, with computers in the home, and in other outside activities (e.g., computer camps). In 
addition, students were asked to report previous experience in the BASIC programming language 
as well as other programming languages. 

Homework 

Most of the nine Metaoourse lessons included woricsheet problems for students to complete 
in class and homeworic assignments to be completed outside of class between lessons. Ths taste 
of sending honneworic and woricsheet papers to ETC was rotated among the teachers so that, for 
each lesson, different teachers were asked to retum their students* papers. These were xeroxed 
and returned as soon as possible to the teachers. 

Teachers returned 83% of the sets of student homewortc and woricsheets asked for (25 out of 
30). Teacher response was better toward the end of the Semester than the beginning. Each 
students homewortc was "graded* and the total number of students making a particular number of 
errors on each problem was recorded. 

Coding Procedures 

Coynrttyft Protest^Posttest. The cognitive test included several types of problems. The- 
daysK)f-the-week problems and the students-and- professors (fomnuta) problem were scored 
either 0 for corTect,or 1 for incorrect. In the triangle-count problem, the correct answer was scored 
0. If the student reported more than the correct number of triangles, the score was the excess 
with a minus sign; if fewer, the score was the shortfall with a phis sign. 

The descriptk)n-of-geometric*figure problem and the repeats-and-decides problems 
required special conventtons because of the complexity of possible responses. In the geometric 
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problem, the following categories were scored: 1) right figure kept, 2) geometric shape omission, 
and 3) position omission. If the correct figure was kept the student scored a 0, and one point was 
added for eacii shape in excess kept. For the shape category, the siudent scored a 0 for correct 
descriptton of shapes, 1 if one shape was omitted, 2 if both shapes (square and rectangle) were 
omitted. If the correct positioning of the squares was correct the student scored a 0, 1 for one of 
the relattonships was missing, 2 if both relattonships (top-bottom/spread out) were omitted. 

in the repeats-and«decides problem, fomi, order and content were defined as scoring 
categories. Egmi referred to the writing of commands in the correct fdmnat. Order referred to the 
conrect ordering in a sequence of commands. Content concerned whether the student's repeats 
and deckles l>rogram*' wouki do the assigned task. All categories were scored 99 if the student 
dki not try the probiera Othenwise the categories were scored on a 0 to 4, where 0 designated a 
perfect answer, 1 , a single enror and so forth up the scale to 4 whtoh indtoated insufficient data/ 
misunderstood task or for the form category, repeats and deckles not used. 

The Attributton of Intellectual Responsbility scale was scored in the usual manner in temis of 
whk:h alternative (internal or external tocus of control) was chosen on the 5 positive and 5 
negative intellectual situattons. 

BASIC Test . The Students' perfomiance on the BASIC test was scored in a number of 
different categories. Many of the categories only suited certain problems, for instance, the 
counter category only applied to problems that included a counter. Each of the 1 6 problems was 
scored for enrors by noting whether or not a studenrs response included certain specific features. 
Responses to Problem 1 (PRINT-string), for example, were scored fbr two features, or error 
categories. To be error free, a response needed to inckide a conrect PRIMT command (the first 
error category) as weH as a quoted string (the second category). The sample response (1) below 
wouM be scored as having 0 errors - it is correct Response (2), however, wouk) be scored as 
having 1 error; the PRIMT is correct, but the quote category is scored as incorrect because the first 
quotatton marie is missing. 

(1) PRINTIhe answer is car. 

(2) PRINT the answer fe car. 

Some problerm called for an "extraneous statements" category. This was scored when an extra 
statement or set of statements was added to the program, whtoh may or may not have resulted in 
an enror, but showed deariy that the student had an improper model of machine semantk^s. 
Several other categories corresponded to the use of various program constmcts; for instance the 
posittoning of PRI^4T statements, R3R-NEXT statements and the like. The number of error 
categories scored for a problem ranged from one in problem 9 (the filhin-the-blank GOTO 
problem) to ten in problem 16 (the complex pitxluctton problem). 

Scnrinq ProcGduTB. Using the scoring system described eariier, 4 scorers on the cognitive 
test and 2 scorers on the BASIC test wortced Independently to code the students' responses to 
the test, crosschecking periodteally with one another and establishing polteies to clarify the 
scoring system. The coding was blind with respect to whether the response came from the 
treatment or control group, or from a pre or post-test. While every student response was not 
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coded by all scorers, the scorers all coded and discussed a random subsample. Policies that 
could affect prtor scoring were applied retroactively. 

After about ten pertdnt of the data had been scored the scorers deemed themselves to be 
adequately calibrated. The remainder of the coding provided the basis for calculating interjudge 
agreement Disagreements were discussed and resolved, though original scores were retained 
for calculating the agreement. All the coding was used for the data analysis in other respects, on 
the grounds that divergences had been discussed and resolved and principles applied 
retroactively to the coding before the calibration. These procedures resulted in very high 
interjudge reliabilities on both tests wth correlations of .90 or greater in all cases. 

In order to detennine whether any of the judges tended to score significantly higher or lower 
than the others, the mean score and standard deviation for each judge were calculated and 
compared for the totals on both the BASIC and Cognitive tests. In each case ANOVA indicated 
no significant differences among the judges. Therefore, for those tests that were multiply scored, 
the judges scores were averaged, while for those tests that were scored by only one judge, that 
score was used in ait analyses. 

Results 

Rriftlitv to MetacoursQ Lessons as Written 

Of critical importance in evaluating any educational inten/ention are questions related to how 
it is actually inpiemented. We obsen^ed a number of Metacourse, and non-metacourse lessons 
taught by each teacher in order to assess how the teachers and students responded to the 
materials. We found that the teachers covered the content of the Metacourse with good fidelity, 
usually rated around 5, where 7 was described as teaching the material ''almost exactly as written" 
and 4 indicated Ihe sanie or very similar content of lesson as written, but adapted and 
paraphrased*. This was consistent with the rating obsen^ed in our preliminary study, also about 5. 
It appears that the lessons were either paraphrased or presented in a manner very similar to how 
they were originally written. 

Teachers were usually rated as covering three quarters of the material in each lesson quite 
adequately and also as referring appropriately during the Metacourse lessons to the major 
principles stressed in the Metacourse, ie. the paper computer, purpo;;e, syntax, action, patterns, 
with the exception of the minimanual which was very rareiV mentioned. Thus the minimanual may 
be an underutilized resource, at least in temns of Ks integration into the instmctors' presentations. 

In contrast to the generally good coverage of the lessons and principles of the Metacourse, 
teachers had much more difficulty in pointing out bridges or applications either to other academic 
discipiines or problems in real life. Such bridges were noted in only about 10% of the classes, 
whether they were Metacourse or non*Metacourse lessons. 
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Hnmftwork 

The primary purpose for the homeworicAworksheet analysis was to determine whether or not 
teachers and students were actually able to use the assignments. The analysis indicated that they 
were. Students seemed to find the assignments appropriate and of reasonable levels of difficulty. 

The secondary purpose was to detenmine where the Metacourse lessons might be 
improved. In general, the homework analysis was most useful in detecting problems, not 
strengths, in the Metacourse and was best interpreted in conjunction with teacher comments. 
This analysis, along with teaci^sr comments was used in preparing notes to be used for teachers 
in a future study in whk:h potential problems in implementing the Metacourse could be 
addressed. 

Some specific problem { noted during the analysis included the following: (a) a number of 
students had difficulty hand oxecuting simple programs that included PRINT and LET statements 
containing expressions; (b) many students did not adopt the pointer that the Metacourse 
suggested they use to keep track of the current Sine during hand execution; (c) PRINT/INPUT 
combinations confused many students - the PRINT statement preceding the INPUT statement 
often interfered with hand execution of the INPUT statement; (d) the Assume User Types area of 
the computer worid was not used by some students, pertiaps because this was not expricitly 
modeled by teachers; (d) many students dU not attempt to klentify patterns in problems, 
presumably because they dki not understand this concept; (e) a common hand execuiton 
problem with somewhat more complicated programs was to miss one of the many required steps 
(e.g.. changing a variable in the Variables Area of the Computer Worid each time it is encountered 
in a FOR/NEXT loop). Many (A these same diffksjities manifested themselves later on in the 
BASIC end of semester test 

DiffflrAnftflfi betwaen Mfltarmiroft and Qrriinarv Lessnns 

As in our previous study, obsen/attons from the experimental sites were examined for 
differences in student and teacher behavior when Metacourse lessons were taught as compared 
to when ordinary lessons were taught. Of partk»ilar interest were any major differences in what 
went on in class during such lessons and whether these seemed disruptive or beneficial. Since in 
each case the observations of a particular class v;ere made by a single obseiver who was not blind 
to the type of dass being conducted, tiiesa results must be interpreted with cautton. However, it 
shoukl be noted that the observers were generally quite experienced, the variattons reported 
between teachers were not large, and the general pattern of results obtained appears quite 
reasonable. 

There were few surprises in the results we obtained. As in our previous study the teachers 
seemed somewhat nwre comfortable when presenting their own lessons (mean rating of 6.9) 
compared to the new Metacourse material (mean rating of 5.8). However both scores are quite 
high (7 point scale) indicating general comfort witti both old or new material. As before, we found 
large differences between teachers' styles, with non-Metacourse lessons rated as almost twice as 
interactive (60% vs. 32%), while somewhat less presentatfonal (26% vs. 15%) than Metacourse 
lessons. During Metacourse lessons the data suggest that more time was spent in lecture (25% 
vs. 12%), demonstiations (20% vs. 11%), and in hand execution exercises (21% vs. 9%), and 

ERIC ' - 18 



14 Programming 



less time in discission (23% vs. 40%). and "other exercises (18% vs. 48%). Student behavior 
during t3oth types of classes was rated as quite responsive, attentive, engaged, and interactive 
with their teachers (ratings generally over 5 in txsth types of classes; however, there was 
apparently more interaction among students during the non-Metacourse lessons (mean rating of 
5.4 vs. 3.4). No dtfferinces were obsen/ed in temis of the nature of such interactions, with both 
teachers' and students' interactions descrfljed as processKsriented and product-oriented about 
half the time, with very little disciplinary interaction occurring at any time. Students seemed quite 
prepared for either type of lesson (5.1 for the Metacourse, and 5.7 for non-Metacourse lessons). 
Rnally, despite the differences that were noted, the overall effectiveness of each type of lesson 
was judged quite high (5.4 for Metacourse, and 6.0 for non-Metacourse lessons). 

Group Differences 

The findings above seem to indicate that despite differences between nomial and 
metacourse lessons, the Metacourse materials were adequately covered although both the 
observations and teacher self reports indicate considerable difficulty in making connections 
between the material and other academic and life problems. 

However, assessing the impact of the Metacourse also requires equivalence of the treatment 
and control groups on other factors that might influence programming achievements, or statistical 
conrectlons as needed. Consequently the data were examined by group to check for significant 
differences extant before the treatment. 

Previous Exparience with Corttputers and Corryuter Programming . Most of the students in 
both groups had little previous experience with programming languages. Three quarters of the 
experimental group and 84% of the control group had no prevk>us exposure to BASIC. Those 
few with experience, however, had minimal exposure (typk^ly a week or two here or there). While 
24% of the experimental group had some experience with another programming language 
compared to only 5% in the control group (p<.05), this consisted iri all but 2 cases in each group 
of a brief exposure to LOGO, some years back. Rnally a little less than half our subjects in each 
group (44% in the experimental and 49% in th9 controls) had a computer in their home. Thus with 
respect to these ^ypes of experiences the groups seemed not substantially different. 

Co gnitive Pre-Test . The cognitive test was assumed to be an indteator of general ability that 
might relate to later BASIC perfomtance, and in fact signifk:ant positive correlattons were obtained 
In both groups ( r«.69, p<.01) in the expenmental group and (rn.40, p<.01), in the control group. 
The Analyses of Variance Indicated no signifteant diflerence in homogeneity of stopes between 
the two groups (F« 1.345, n.s.). 

Overall the control group perfomted significantly better than the experimental group (p<.Ol), 
commiting on average about 12 errors compared to 16 errors for the experimental group. The 
range was from about 6 tc 29 errors in the 9 experimental cla&'ies compared to 5 to 1 6 errors in the 
13 control classes (A more detailed analysis of subtests is presented later when the issue of 
transfer is conskierad.) 
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Thus the control group appears to be of somewhat higher ability than the experimental 
group, yet as the data will indicate the experimental group perfomied significantly better on the 
BASIC test administered at the end of the course. 

Impart of Instniction on Mastery of BASIC 

Tablel (next page) indicates the perfonnance of experimental and control groups in terms of 
mean errors on each of the 16 BASIC Test problems as well as subtotals for four types of 
problems and the total test. Despite the apparent higher general ability level in the control group 
the experimental group averaged about 5.5 fewer errors, nearly half a standard deviation overall, 
than the control group. They prockjced about 77% correct responses compared to about 66% for 
the controls (a maxinrum of 61 errors was possible). 

Furthemiore, the experimental groups dio significantly better than the controls on all major 
categories of problems, with the smallest advantage ocming on the simple one line command 
problems and the production problems (about a thiro of a standard deviation), and the largest on 
the hand execution problems (nearly two thirds of a standard deviation). 

Results for individual problems within three of the major categories ~ Hand execution. 
Debugging and Production - show that the experimental group had significantly fewer errors on 
every problem except one requiring the use of art IF/THEN statement to create a loop 
(ProblemiS). In fact, both groups found it more difficult to write a program using IF/THEN after 
being asked to create a program that generated identical results but used the more natural FOR 
loop (Problem 14). 

Interestingly, there was also no significant difference between the two groups on the 
IF/THEN problem within the other major category -* Simple one line commands. The only 
problems within this category for which there were significant differences included one which 
asked students to set a variable to a string (Problem 6), one requiring the completion of a 
FOR/NEXT loop (Problem 8) ind one in which students needed to recognize when a GOTO 
statement was needed (Problem 9). These last two problems (8 and 9) differed from the other 
problems within this category because partial code was provided that, presumably, needed to be 
read and understood before being completed. In this respect, periiaps, those problems were 
similar to the Hand^xecution problems. 

The BASIC test problems described above often contained the same features or enor 
categories. This allowed us to undertake a finer analysis of how students handled a pa.:!cular 
feature across and within various problems. These additional error categories are described 
betow, and the results presented in Table 2. 
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Table 1. Errors (Mean Number) on Each Problem of the BASIC Test 



Simple One Line Comma nri Problflms 





1 


2 


3 


4 


5 


6 


7 


8 


9 


Total 


Exp. 


.13 


.13 


.53 


.11 


.25 


.71 


.88 


.66 


.27 


3.67 


Cont. 


.08 


.11 


.72 


.16 


.22 


1.14 


.88 


1.13 


.49 


4.92 


Diff. 


.05 


.02 


-.19 


-.05 


.03 


-.43*** 


-.00 


.47*** 


-.22*** 


1.25*** 



D/S.D .32 



Hand Executton Problems DeBugging Problem 





10 


11 


13 


Total 


12 


Exp. 


.28 


.74 


1.65 


2.67 


.61 • 


Cont. 


.73 


1.47 


1.94 


4.14 


1.05 


Dif. 


-.45*** 


-.73*** 


-.29* 


-i.4r** 


-.44*** 


D/S.D. 






.61 




.48 



Production Problems Ali ProblftmS 





14 


15 


16 


JsM 




Exp. 


1.13 


2.83 


3.41 


7.36 


14.31 


Cont. 


1.60 


3.11 


5.07 


9.77 


19.88 


Diff. 


-AT 


-28 


-156* 


-2.41* 


-5.57*** 


D/S.D. 








.34 


.45 



Experimental Group (n«1 20) * p<.05 

Control Group (n»224) ** P<.01 

*** p<.001 
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SImniB ona Una commgnd flrrnr cateoorias f PtPblems 1^9): 

1. PRINT (Probiems 1-2): The word PRIHT is not present or is spelled incorrectly. 

2. Quote (Problems 1 and 3): The string following the PRINT command is inconect or is 
not preceded by a quot^tt'on mark. 

3. LET (Problems 4 - 6} : The word LET is not present or is spelled incoaectly. 

4. Assignment (Problems 4 - 6) : Tna assignment statement following the LET command 
is incorrect. 

5. ACTION (Problems 7 - 8): Tne expression following THEN in problem 7 is incorrect; 
the expression between the FOR and NEXT commands in problem 8 is incorrect. 

6. GOTO (Problem 9): The GOTO command or the line number following it is incorrect. 

7. INPUT (Problem 3): The word INPUT is not p- esent or incorrect. 

8. M ISC. (Problems 2,3,7,8): Various enrors occuring in only one problem. 

Hand.flxft«ition ftrmr eateaorias f Problems 10, 11 and 13): 

9. Prompt/ir^ (Problems 10, t1 and 13): The oittput generated from INPUT 
statements does not include the correct prompt and/or input. 

10. Output (Problems 10, 1 1 , and 13): The output generated from PRINT statements is 
incorrect. 

1 1 . Minor execution: Any execution enors not covered by other categories. 

1 2. Format: Output is not property fonnatted.Debugging enor categories (Problems 1 1 , 
12, 13): 

13. Bug-1 1 (Problem 11): Jiia correct ("ui.jxpected^ output is not generated. This 
category does not include categoYies Vke minor execution, output and so on that 
were also scored in .'^roblem 1 1 . The only feature considered is the one directly 
related to debugging. 

1 4. Bug-1 2 (Problem 1 2) : The correct INPUT statement is not added to the program (as 
above, other "non-bug" categories are excluded from this category). 

15. Bug-13 (Problem 13): The correct ("unexpected") output is not generated (as in 
category 13, other "non-bug" categories are excluded). 

Production ermr ftateqories /Problems 14, 15 and 16) 

1 6. Counter (Problems 14 and 15): The counter following the FOR command is 
incorrect. 

17. Action (Problems 14 and 15): The major action (PRINT X) is incorrect. 

1 8. FOR (Problems 14 and 1 6): The FOR statement is missing or incorrect. 

19. NEXT (Problems 14 and 16): The NEXT statement (NEXT X) is incorrect or missing. 

20. Variables (Problems 1 4 and 1 6): Incorrect number or type of variables used. 

21 . Extraneous (Problems 1 4, 1 5 and 1 6): The program includes unneeded commands 
that interfere with the mnning of the program or that make the solution clumsy. 

22. Minor syntax (Problems 14, 15 and 18): This category Includes enors not covered 
by other categories that ocur in more than one problem. 

23. Misc. (Probk>n>s 15, 1 6): Various error types appearing in orjiy one problem. 

Table 2 (next page) presents results from these indivkiual enor categories within and across 
problems. Like the results presented in Table 1 , these results indicate that the Metacourse 
group made significantiy fewer errors in each of the four major categories. However, they also 
indicate that this dki not occur unifonniy within each major categciy- That is, the experimental 
group, although never commiting significantiy more errors than the comrol group, did not perform 
significantly better than the controls on all error.categories. 
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TABLE 2. Types Of Errors (Mean Number) on the BASIC Test 



SImnte One Llna Command Errors fProblems 1-9) 

123456789 
PRINT QUOTE L£I ^SSm i^^:!!^ ^aQIQ INPUT msSL TOTAL 

Exp. .04 .23 .45 .64 .70 .27 .21 1.14 3.67 
Cent. .05 .22 .80 .72 .93 .49 .35 1.36 4.92 
Diff. -.01 .01 -.35*** -.08 -.23** -.22"* -.14" -.22 1.25" 



Hand Execution Errors f Problems 10.11.13) 





9 


10 


11 


12 


13 




PROMPT-INPUT 


OUTPUT 




FORMAT 


TOTAL 


Exp. 


.70 


.13 


.53 


.M 


1.40 


Cent. 


1.87 


.08 


.60 


.06 


2.60 


Diff. 


-1.17*" 


.05 


-.07 


-.02 


-1.20"* 



Dabug^lnq Errors fProhlflms 11.12.13^ 





13 


14 


15 






BUG-11 


BUG-12 


'•SUQ-13 


TOTAL 


Exp. 


.48 


.26 


.78 


1.53 


Cent. 


.76 


.48 


.78 


2.02 


Diff. 


-.28*" 


-.221"* 


.00 


-.49"* 



Production Errors (Problems 14.15.16^ 





16 17 


18 


19 


20 21 22 23 






COUNT. ACTION 


£QH 


NEXT 


\im. DaBA.MIN.SYN. MISC. 


TOTAL 


Exp. 


.48 .42 


.32 


.34 


.52 1.02 . 96 3.30 


7.36 


Cent. 


.58 .57 


.64 


.76 


.74 155 1.16 3.98 


9.77 


Diff. 


-.10 -.15 


-.32"* 


-.42"* 


-.22" -.33" -.10 -.68 


-2.41" 



Experimental Group (n-1 20) * p<.05 

Control Group (n«224) " p<.01 

"* .p<.001 
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Wtth ra$pect to the -simple one line command errors" (In the first 9 problems) the 
experimental group did no better than the controls on a number of simple error categories such as 
PRINT and quote and assigning a value that a variable should take; however, they made about half 
the errors of the control group when producing simple LET statements (error category 3) and 
about one quarter less errors than the controls in specifying the action of simple IF/THEN 
statement or FOR loop (category 5). 

As noted above, the experimental groups made significantly fewer hand execution errors on 
problems that required creating a screen display when given a program (Problems 10, 1 1 and 13). 
It appears that the experimental groups* superiority is due to its ability to represent what a prompt 
and input statenr)ent look like on the screen (Error category 9), since there were no differences in 
the output , minor executton and fomiat error categories (10,1 1 ,&12). 

If we In turn ignore the pravtous error types that may occur during hand execution and look 
speciftoaliy at the aspects of executton that are directly ralated to program bugs, we find that the 
experimental group is signiftoantiy better at detecting unexpected output (Error category 13 from 
the Garden Path problem). They are no better, however, at detecfing the parallelism bug 
(Category 15). Bug-12 is pertiaps a more reallstte debugging problem. The students were given a 
program and sinfY)ly asked to correct as weU as find a bug. In this situatton, the experimental group 
was again signiftoantly better than the non-metacourse group. 

The final set of error categories to be conskiered are those related to the productton of 
whole programs. When given three programming problems, the group with Metacourse training 
handled FOR/NEXT loops better, was better able to choose the con^ct number and type of 
variables to use and produced significantly fewer unnecessary statements (error categories 18 
through 21). There were no signftoant differences between the two groups on the remaining 
production ennor types. As was the case with all other variables analyzed in this study, the control 
group did not make signifk:antty fewer errors in any of the categories investigated. 

Transfer 

As noted in our prevkxjs reports, while the cognitive posttest provided an occaston to 
examine possible transfer of cognitive skUls from programming instnjction it should be recalled 
that such a finding was not necessarily expected from the Metacourse treeatment. Most research 
on the impact of programming Instructton on cognitive skills has not found substantial gains, 
except under very special condittons (ie. specifk; emphasis on transfer throughout a program, 
one*on-one tutoring, and tests composed of problems formally similar to those encountered in 
the programming course). In contrast, the Metacourse was designed specifically to advance 
programming skills rather than to promote general transfer. Nonetheless, the data presented an 
opportunity to exptore the issue within this limited context. 

Impact on General Cognitive Skills 

As indk^ated prevkHJSly the general cognitive tests were composed of 6 separate subtests 
(See Appendix D). There was almost no variability of the Attributton of Intellectual Responsibility 
subtest. Both experimental and control groups attributed an average of betwen 7.7 and 7.9 
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questions (out of a possible 10) to internal factors, both on pre and posttests. Analysis by positive 
or negative outcomes yielded similar results. Thus most of our sample attributed their successes 
and their failures primarily to internal causes, both prior to and after the BASIC programming 
course. This variable was therefore excluded from all further analyses. 

Table 3 (next page) presents the perfonnance of each group on each of the other 5 subtests 
and their total score for both pre and posttests. 

As indicated previously the control group made significantly fewer enors on the pretest 
overall than the experimental group (p<.00l). Analysis by subtests ir<dicated that this superiority 
manifested itself on three of the five subtests (conditional reasoning, precise visual description, 
and repeats and decides), while no initial differences were found on the visual planning or 
professors/students problems. 

Surprisingly the expeilmental group improved by about three fewer enors on the posttest, 
compared to virtually no gain for the control group. It Is unlitely that this highly significant outcome 
(p<.(}01) would be due merely to regression to the mean. Analysis of the data from the 
component tests indicates that this result is due almost entirely to the considerable improvement 
of the experimental group on the Repeats and Decides subtest, compared to a modest decline in 
perfonnance on the part of th« oomr^i group on that same problem. Thus some evidence of 
transfer was obsenrad, but it oocunred only on the problem most closely related in its fomnal 
stnicture to that of producing coherent conrands in a programming language. 

Taachars' Raarttons to the MatacQurse 

Since this study was conducted within the laboratory sites context described eariier, we 
received frequent feedback from the teachers concerning the metacourse materials. The 
teachers were in general quite positive in their evaluation of the revised metacourse. The utility of 
the visual model of the computer worid, the purpose, action, syntax analysis and the emphasis on 
patterns received the most praise. The length of some lessons, and the difficutty of integrating 
these long, rather rigid lessons smoothly into their normal course were most often cited as 
negative features. Many teachere indicated that they wanted more freedom than the tightly 
controlled conditions of the experiment permitted. It was apparent that infusing new concepts 
into their normal lessons is a skiU not quicKiy masteied, and while we would expect easier 
integration "the second t!me around," future interventions need to address these concerns. 

Discussion 

The targe scale study reported here was undertaken in order to assess the effects of a 
"metacourse" which was designed to enhance novices' learning of BASIC by provkling a small 
number of key concepts and strategies that, our clinical research revealed, many students fail to 
acquire on their own (Peri<ln8,.Hancock, Hobbs, Martin, & Simmons, 1986; Peridns & Martin, 1985; 
Peritins, Martin , & Farady, 1986; Perkins, Farady, Hancock, Hobbs, Simmons, Tuck, & Villa, 1986). 
A pilot study reported in Peridns, Farady, Simmons & Villa (1986), employing an eariier version of 
the metacourse, although encouraging, produced IHtle hard evidence on its effectiveness. It was 
hoped that the current study would be able to provide stronger evidence on the issue. 
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TABLE 3. Errors (Mean Numbar) on Problems t.^ the Cognitive Pre- 
and Post-tests. 



Problem Tvaa 

Cnndrttonal Roasonino Visual Planning 





Pretest 


Posttest 


Gain 


Pretest 


Posttest 


Qsm 


Exp. 


1.71 


1.56 


.15 


6.24 


4.92 


1.32 


Cent. 


1.36 


1.33 


.03 


5.26 


4.81 


0.45 


Diff. 


.35" 


.23 


.12 


.98 


.11 


.87 



Problem Tvds 

Professors/Students Precir^Q Visual Description 





Pretest 


Posttest 




Pretest 


fj2SttQSi 


Gain 


Exp. 


.72 


.76 


-.04 


2.87 


2.63 


.24 


Cent. 


.68 


.54 


.14 


2.02 


2.03 


-.01 


DW. 


.. .04 


.22*" 


-.18* 


.85** 


.60* 


.; .25 



Problem TvpQ 





Reoeats and Decides 




Total 






Pretest Posttest 


Gain 


Pretest 


Posttest 


Sain 


Exp. 


4.84 3.40 


1.44 


16.34 


13.22 


3.12 


Com. 


2.95 3.53 


-.58 


12.34 


12.29 


0.05 


DIff. 


1.89*** -0.13 


2.02*** 


4.00*** 


.93 


3.07* 



Experimental Group (n-122) 
Control Group (n«219) 



* p<.05 
'* p<.01 
'* p<.001 
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As in our previous report it is appropriate to consider the three questions: Did the 
Metacourse prove "teachable", leading to an implementation that smoothly provided the intended 
concepts and practice in their use? Did the Metacourse have the hoped-for impact on students' 
mastery of programming? DkJ the Metacourse, or the normal programming instruction in the 
control grx)upsi have a cognitive impact beyond the targeted instruction? Rnally questions of 
possible long term effects, and future directions should also be briefly considered. ^/ 

Teaching . Once again the data indicate that at least for experienced teachers under 
conditions of conskJerabie support the Metacourse was quite teachable and could be integrated 
into the normal cunriculum in BASIC. Although the Metacourse lessons produced a variety of 
differences in teacher style of presentation as well as student beiravlor, compared to the normal 
lessons, both teachers and students appeared to adapt well to the new material with a number of 
Indicators pointing to effective classes. Further it may well be the case that some of these 
differences in observed behaviors (e.g., anwunt of student-teacher interaction) might be due to 
the fact that this was the first time the teachers used the materials, and that such differences may 
diminish as teachers get more familiar with the material. 

The evidence on ttie extent to which the instaictors were able this first time through to 
"infuse" the key Metacourse concepts into their entire course is not as clear. We did find that 
during the Metacourse lessons teachers referred to eariier key concepts quite often, but the 
results are not clear from the limited number of non-Metacourse lessons we coukJ observe as to 
how often these concepts were used in regular classes. We dkJ note a general absence of 
bridges or applications of these concepts to any areas outside of the programming problems 
themselves. It seems clear that deliberate efforts have to be orchestrated if such bridges are to be 
built 

Impart on proyrammtnq mastery . It is clear that our oxperimental groups evidenced a 
considerable advantage in their general competence as beginning BASIC programmers 
compared to the control groups. Further, the results Indicate that this improvement is manifested 
on a variety of programming skills, such as: use of correct syntax to perfomn simple operatksns, 
ability to trace the acttons of a program through hand executton, and the ability to debug and 
produce simple programs. While the Metacourse did not produce enhanced perfomiance on 
every problem, e.g., detecting the "parallelism bug" (Pea,1985), It dkJ produce fewer errors on a 
variety of signifk^nt problems typteal of those encountered in an introductory BASIC course. 
What is not so apparent is the cvhsb of the improved performance. A number of possibilities are 
plausible, and the results may well be due to a combinatk}n of factors. 

We woukj hope that the Metacourse with its emphasis on the students' devekspment of 
mental models through whfch they understand what the computer does, strategies by whteh to 
organize their problem-solving efforts, etc. played a signiffcant role. However, It must be 
acknowledged that the "laboratory sites" interventksn of whk:h this study was a part was an atypk:al 
treatment in that the partfcipaling teachers received much more support than is nomially the case 
when a new curriculum interventfon is Introduced. Further, we couW not randomly assign 
teachers to treatments. The laboratory site high school programming teachers were all required to 
use the Metacourse, while the control group was fomned by solteiting vohinteer teachers at similar 
high schools. While the teachers in both groups were experienced teachers of BASIC, it is 
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possible that our treatment teactiers were simply a group of exceptionally talented instructors who 
could produce large gains even with a treatment with which they were unfamiliar. One should 
note that our study, like most in the field, pits teachers using a treatment program for the first time 
against controls who employ their "nomfial** curriculum which they have typically worked through 
many times. 

Another study, currently in progress, shoukl help resolve some of these ambiguities. In this 
study the revised Metacourse is being used by 9 other teachers in 7 new high schools under 
conditions that more nearly duplicate nomnal classroom innovations. That is, these teachers have 
been given the Metacourse along with some "Metacourse memos" (notes to teachers on 
experiences teaching with the Metacourse), and are provkled with no other supports. The same 
assessment instrum3nts have been employed as in the previous study. The results from this 
study shoukl he^ clarify the signiftoant factors inf luendng the improved peribmiance in BASIC. 

Transfer Although our results were in general consistent with most of the literature, in that 
there was little evkJence of transfer either on rTX)St cognitive tasks or on our affective measure, we 
dkJ obtain evklence of transfer on one subtest that was stnictually somewhat similar to a typical 
programming task. It may also be the case that an affective instmment more spedfk^ally focused 
on students attitudes concerning errors may yet reveal some changes in this realm. 

As we have pointed out prevkxisly, we believe a kind of tradeoff may exist between teaching 
for programming corTf)etency and teaching for transfer of cognitive skills from programming 
(Pericins, Schwartz,& Simmons,in press). The metacourse as cunrently designed still focuses on 
the development of programming competency rather than transfer of general cognitive skills. We 
provkie little that does not have direct bearing on programming competency. The next section 
discusses some future directtons that may affect this balance. 

Future Dlrocttons, While the results obtained in this study are encouraging, even the 
perfonnance of the treatment groups on our BASIC end of semester test left consklerable room 
for improvement (77% correct responses). Further, teachers comments made it dear that, though 
valuable, the Metacourse in its current 9 lesson format makes both the integratton into their normal 
cunlculum, and the infusion of key concepts throughout the tenn, a conskierable challenge. 
Students still often exhibit an inert, rather than active knowledge of the programming enter^nse. 
This *1ragile" knowledge does not permit optimal use in contexts when it should be app .i/en 
within the disc^line of programming let atone in other problem solving reBiws. The prr , amming 
research group at ETC is currently devetoping a ''second generatton metacourse", to address 
some of these concerns. 

This second generatton metacourse is built upon a "language" rather than a "lesson" vnodel 
Thus rather than a collectton of fully worked out lessons, a number of short modules introduce 
the key concepts, while other modules focus expltoitly.on how such concepts can be practiced or 
"infused" throughout the sen)ester. Rnaliy a number of optional modules attempt to illustrate how 
bridges can be made to academto and "real life" problems, thus encouraging transfer to more 
general cognitive skills. While provkling teachers more freedom to utilize the Metacourse in their 
own fashton, it is designed to make more explicit and hopefully available to both teachers and 
students the core elements of our approach and the rattonale behind them. Such aids as an 

er|c 



24 Programming 



"animated computer vworid" with an agent perfomiing simple operations on infomiation, icons, 
metaptiors, and posters are ail tseing considered as metacognitive supports. This second 
generation language model metacourse will be empirically tested in the near future. 

What has become more apparent as we have interacted with teachers is that many of them 
share a goal of teaching their students In some sense "what computers and programming 
languages are really all about", "how they can be used as problem-solving tools" and not just the 
syntax of a particular programming language Ike BASIC. It may be the case that the achieve:nent 
of this goal through the design of a second generation metacourse is also mora compatible with 
the development of general cognitive skills, and that a powerful pedagogy of programming 
focusing on attention to mental models, strategies, etc. may lead to more success at achieving 
both programming competence and transfer. 
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the progratn* (c) Change the * X print statement so that It 
explains the output. 

The Computer World 



Program Area 



PRINT "HOW MANY PEOPLE?" 



INPUT P 



G = 0 



PRINT "PLEASE ENTER THE NUMBER" 



PRINT "OF SLICES OF PIZZA THAT 



PRINT "EACH PERSON WANTS" 



FOR I = 1 to P 



INPUT N 



If N > 3 THEN G = G + 1 



NEXT I 



PRINT "THE ANSWER IS";G 



HIM 

C" 0 

V V 



^ A 




'□DOODDDDDDDDDDDDDDDtt 
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Example of Purpose, Action, and Syntax 
from Mini-Manual 
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PURPOSE: You use LET in a program to make the computer save information m 
its memory for later use. The information is stored m a variable. Think 
of a variable as a box with name. Boxes with names ending in $ hold only 
strings. Boxes without the $ hold only numbers. 



ACTION 



SCREEN DISPLAY WORK SPACE 



SYNTAX 

LET Variable Name = Number 

for examole? 
LET X = 5 

Is it really there? 
PRINT X 



LET Variable Name = Legal Numerical Exoression 




5 is stored m the X dox 



for examolei 

LET Z =^ 5 ( 1 + X ) 

Is it really there? 
PRINT Z 



LET Z = Z - 2 



Is It really there? 
PRINT Z 




1 + 5' is 6 

5*6 gets vou to 30 

30 is stored m the 2 box 



30 - 2 = 28 

2S IS the new value for Z. 
which IS stored in the Z box 



LET String Variable Name » Strxno 
for examplet 
LET A> = "SIZE" 

LET B* = V* 

Are they really there? 
PRINT A% 

PRINT BS 




SIZE is stored m the A$ 
box. 

/ IS stored in the B$ box. 



Note: You can leave off the LET statement, and get the same result, 
for example: 

X = 2 2 is stored in tne X box. 



Df = "DAYS" 
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DAYS is stored in the 
D$ box 



Appendix C 

Example of a "Pattern" from 
the Mini-Manual 
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Summing Pattern 

Purpose: To find the sum of a series of numbers 
Structure: Example: 

10REMADDS5NUMBERJ> 
LETT^O 20LETT = 0 

<begin loop> 30 FOR I = 1 TO 5 

40 PRINT "ENTER A NUMBER" 
50 INPUT N 

LETT = T + <numbert0 8dd> 60 LETT = T + N 

<enciioop> 70 NEXT I 

Action: The summing variable (in this case, T) is first set to 0. Each time 
the loop body is executed, another number is added to the summing variable. 
When the loop is exited, the summing variable contains the total.. 



Conntinq Pattern 

Purpose: To count the number of times something is true 
Structure: Example: 

10 REM COUNTS A'S 
LETC = 0 20LETC=0 

*<begin ]oop> 30 FOR I = 1 TO 5 

40 PRINT "ENTER A GRADE" 
50 INPUT G$ 

IF <condit1on> THEN LET C = C + 1 60 IF G$ = "A" THEN LET C = C + 1 
<enci loop> 70 NEXT I 

Action: The counting variable (in this case, C) is first set to 0. Each time 
the loop body is executed, you test a condition. If that condition is true, the 
counting variable is increased by 1. 
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Cognitive Skills Pre-/Post-Test 
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Version A 

Date: 



Name: 



GEMERAL IMSTRUCTIOMS 

Our class has been chosen to contribute to some very important educational 
research. A group of people at Harvard University is attempting to discover 
the best techniques of teaching BASIC programming so that it will be more 
interesting and understandable to students. 

This post-test, given now at the end of the course, will help them to 
determine the differences among all of you who will be participating. 

The Harvard Group wants you to know how much they appreciate your help and 
they are looking forward to your comments and the wonderful information 
they will gather through your efforts. 

This is a 40-minute test comprised of a short questionnaire and a number 
of problems. 

Do not start the test until your teacher says "go". 

You may work through the test as quickly as you want. After finishing one 
part, go on to the next. 

However, we want to be sure you try all parts. So after the time for a 
particular section is up, the teacher will say, "please go ahead^to section 
A, B, C, D, E, or F (whatever it is) now if you haven't already . 

You can go back to work on a previous section if you have extra time. 
Ask any questions you have now. 
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A. CHOOSE OME ANSWER 5 Minutes 

Pick the answer that best describes what happens to you or how you feel. 
There are no right or wrong answers. 

1. When you read a story and can't remember much of it, is it usually 
a. because the story wasn't well written, or 

b. because you weren't interested in the story? 

2. If a teacher says to you, "Your work is fine", is it 

a. something teachers usually say to encourage pupils, or 

b. because you did a good job? 

3. suppose you weren't sure about the answer to a question your teacher 
asked you, but your answer turned out to be right. Is it liitely to 
happen 

a. because she wasn't as particular as usual, or 

^b. because you gave the best answer you could think of? 

4. When you read a story and remember most of it, is it usually 
^a. because you were interested in the story; or 

b. because the story was well written? 

5. If the teacher didn't pass you to the next grade, would it probably be 
a. because she "had it in for you", or 

^b. because your school work wasn't good enough? 

6. suppose you don't do as well as usual in a subject at school. Would 
this probably happen 

a. because you aren't as careful as usual, or 

b . because somebody bothered you and kept you from working? 

7. If a boy oi a girl tells you that you are bright, is it usually 
a. because you thought up a good idea, or 

b. because they like you? 

8. Suppose you became a famous teacher, scientist or doctor. Do you 
think this would happen . . 

a. because other people helped you when you needed it, or 

^b. because you worked hard. 

9. Suppose you are showing a friend how to play a game and he has trouble 
with it. Would that happen 

a. because he wasn't able to understand how to play, or 

b. because you couldn't explain it well? 

10. If you can't work a puzzle, is it more likely to happen 

a. because you are not especially good at working puzzles, or 
^b. because the instructions weren't written clearly enough? 
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6 Minutes 

n CAYS OF TME HEEK 

, . the d,v to«=-=. i« sundry. «h.t i. th. d.v 

1. If the day before the day a^" 
before yesterday? * 

tomorrow i« what? 

, - fchat is the day before yesterday? 
3. 9..ppose today is the day before Tuesday. «.at 



- r c.ndav vesterday would be Monday 

u..r>. Wednesday instead of Sunday, yesi-e 
A, If tomorrow were weoneaooy 

instead of what? . * 



C. VISUAL PUZZLE 



5 Minutes 





4 Minutes 

B. FORMULA QUESTION ^ 

neoole who drink co^^fee, 1 person drinks ^J^; ^I^PJl nu,.ber 
:t°:nrs%or%rnL:b" of .eople Who ^^^^i i^es" t^e relationship 
r/people WHO drinw tea ^^^-^^^^.^^^^ -"V P-Ple drin. tea. 
between how many people drinK c 



I « C/T 



T » 3C 



C « T/3 



C s 3T 



3 « C - T 
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E. 



DESCRIPTION 



8 tiinutefi 



r ifmmm. «hoHn in the folloning figure. 
Ex.mne crefully ^'^^^ll^^'^^^li^'^'^oll^* Tth«t he/ehe could pick out 
write . description of Iten. -G *,on.. p.per do«n't h.ve the 

-C" from all the other •h.pes. J^*^^„/'he figure. *re •!! mixed up. 
letters beside to identify .Jt*^;^*"'' 
write out 'Our descriptions of "C below. _ 



A. 



B. 



0 O -Q o 

O B 13 o 
0 O 0 O 



C. 




1 



D D P O 

I 1 



O O D 




r. 



I. 



0 O 0 o o 

01 I o 

0 



b 6 o 



J. 




L. 



0 OOP 
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F. REPEATS AMD DECIDES 10 Minutes 

For the next problem, you have to learn about repeats and decides before 
you do the problems. 



REPEAT! A repeat it an instruction to do something over and over again , 
until some condition is met. For instance: 

REPEAT - Jump up and dovm until you've done it 17 times. 

decide: a decide is an instruction to choose which 2n£. of gfi or ^ 
^S^ons exist, and then doing something onc£. based on this choice. 
For instance: 

DECIDE - Decide if you have homework. If you do. stay home and 
do it. If you don't, then go see a friend. 

WRITING DIRECTIONS USING REPEATS AND DECIDES: You can 

using repeats and decides. Here is on example that uses one repeat and 

one decide instruction. 

suppose you want to add up all the money in everybody's pocRet in the room 
and shout "Hurrah!" if the total is over $100. 
Here are the directions: 

REPEAT - Add on the money in the next person's pocket until you 

have covered all the people in the room. 

DECIDE - Decide if you have more than $100. If you do. shout. 
"Hurrah!". If you don't, then be quiet. 



Other problems may require more than one repeat and/or decide. 
Now go to the next page for the problem. 
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It's Saturday and you want to go out and see the movie called: "The Crazy 
Computer". You have a newspaper that lists the movie theaters alphabetically 
with What is playing at each. If you can't find the m«vie -"y^J^ere then 
turn on the T.V. to a ball game. If you find the movie ^^^^ ^^^^^f .J^J'^"^ 
your friends until one says he or she will go with you. If you f nd one 
wiUing to go. then make a date and go. If not stay home and watch the 
ball game. 



If you have extra time, go back and complete sections you did not finish 
or cUsck your work. 
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Appendix E 
BASIC Test 
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NAME 



DATE 



DIRECTIONS — Work quickly but carefully on the following problems. If 
you get stuck on one problem, go on to the next. 



Write the Basic commands that will cause the following to happen. 
An example is done for you below. 

EXAMPLE. The following word appears on the screen: 
Hello 



1. The following message is to be printed on the screen: 
The answei" is cat 



2. The number 5 is to appear on the screen. 



3. The program ashs the user for a number. (The number will be stored in 
a variable. ) 



4. The variable X is set to the value 2. 

5. The variable X is set to 3 more than Y. 

6. A variable (give it whatever name you like) is set to the following: 

THE ANSWER IS KNIGHT 
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7. When the value of 2 is greater than 3. the following message is 
printed: L = 5 



Complete the next 2 programs by filling in the blanl^s with the appropriate 
code. 



8. This program uses a FOR statement to print the number 6 ten times. 
(Fill in the blanks.) 

10 FOR X = 1 TO 

20 PRINT 

30 . 

. • 1^ r.=T¥ of a lareer program. What is needed in this 
l-JT, lH"p:;srai%rrhariintrr2l230^r. not a.ecut... (fiU In t.e 

blank. ) 



210 . 

220 Print "No" 
230 Print "Ho" 
240 print "Yes" 



10. During a run of the following program what will appear on the screen? 
Suppose the user enters 16 for A. 



10 PRINT "HOW OLD ARE YOU?" 

20 INPUT A 

30 IF A=15 THEN PRINT "TOO OLD 

40 IF A>15 THEN GOTO 60 

50 IF A<15 THEN PRINT "LITTLE niD" 

60 PRINT "RETIRE FROM SCHOOL" 

70 END 



screen display 



il. -During a run of the above" program what will appear on the screen. 
Suppose the user enters 14 for A. ^^^^^^ display 



of bread.) 

The program doesn't work. The output is not correct; it is always: 



Number of slices 

Total calories in bread 

0 



What is wrong with the program? Find the error(s) and correct the program. 



100 PRINT "Number of slices" 

110 LET C = S * 70 

120 PRINT "Total Calories in bread" 



130 PRINT C 



140 END 



-n""" "s^i^^ i^TiT'rul i^... . ,o. 

During the run what will tho screen display show? 



10 LET P = H 



screen display 



20 INPUT "HOURS "; H 
30 INPUT "WAGE "; U 
40 PRINT "PAY = "; P 



50 END 
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xou will be writing 2 programs that do the same thing; they Print out on 
the screen the whole numbers from 1 to 10. The output will look like this. 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 



In your first program, use a FOR/NEXT loop. In your second program create 
the loop using an IF statement. 

14. PROGRAM WITH FOR/NEXT LOOP 15. PROGRAM WITH IF STATEMENT 
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16. Each day. a grade school class of 20 students ""^^^ ^^^.J'f"?^^^ 
cartons of milk needed by the class. Write a program that will- J) [°J^_^ 
each of the 20 students ask how many cartons ot miiK ne or .he "i-- - 
and 2) calculate the total number of cartons needed by the class and print 
the total on the screen. 
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Appendix F 
Classroom Observation Sheet 
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1 

Qfi<^ERVATIQN INSTRUnENT 

Observer: ^ 

School: 

Teacher: 

Beginning time: _ 
total class time: 

control: 

experimental: metacourse lesson # 

ordinary class 

taped? yes no 



Date: 



ending timet 



1, Write out an outline of the class as things happen. Try to note the 
essential features such as major points made, method ^f presentation, 
practice time, etc. Revise after class if necessary. 
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Instruction: Pease describe the behaviours indicated below on a 7 point 
scale. For most examples, there are definitions and reference points 
given, reel free to assign ratings between the reference points, but use 
whole numbers only (no fractions or decimal points). If a question is 
unanswerable or somehow not applicable, please mark an "x" in the space. 
(Where fitting, a brief explanation would be appreciated). 

2. Aniount of interaction between teacher and students - 



7 - lots of interaction between students and teacher 

4 - average amount of interaction 

1 - little interaction 

3* Amount of broadly beneficial interaction among students - 

7 - much 

4 - some; a few comments between students 

1 • none 

4. Amount of deleterious interaction among students 

7 * • much 

4 - some; the usual amount of fooling around 

1 - none 

5. Student responsiveness - 

7 - students respond readily 

4 - students need to be coaxed 

1 - students are unwilling to participate, to answer or to 



ask questions, etc. 



Attentiveness of students - 



7 
4 
1 



- highly attentive to material/instruction 

- somewhat attentive, but also talking, etc 

- inattentive — talking, doing other work 



Percentage of students who seem to be engaged in class activities 
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3 



8. Teacher - student interaction ( Give percentages) 

Teacher's role Students' role 

a. Process oriented 

Product oriented 

c. Disciplinary 



9. Preparedness of students - 



7 - students seem very well prepared and ready to move on 

4 - students seem adequately prepared 

1 - students seem to lack understanding requisite to 

continuing 

10. Write in the approximate percentages of the total time 

spent in: 



lecture 

questions and discussion 

demonstration 

hand execution exercise 

other exercises 



(total should be 100^) 



lecture - teacher speaks 

demonstration - teacher demonstrates some aspect of programming 

either on board or with overhead 
hand-execution - students/teacher go over program line by line 
other exercises- 



11. Teacher-material interface - 

7 - teacher seems very comfortable and conversant with 

material 

4 - teacher seems adequately comfortable with material 

1 - teacher seems uncomfortable with material 

12. Mode of instruction 



Give percentages. 

- highly interactive; teacher often asks for comments, 

talks to and with students 

- combination of interactive and presentational 

- presentational; teacher describes, demonstrates, 

models 
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13. Subjective, holistic rating of total effectiveness of 

instruction 



13a. Comment - 



14. Reference to other lessons - 

7 - much reference to earlier lesson(s) 

4 - some mention of earlier lessons(s) 

1 - no mention of earlier lesson(s) 

14a. If ideas from earlier lesson(s) were referred to, note which 



15. Check; (use 1 for "yes" or 2 for "no") Were there any bridges or 
applications stressed for which Basic Programming principles are used in: 
a. other academic disciplines; which one(s): ; . 



b. problems in real life, outside school; which one(s): 



How? Please note: 



For experimental lesson classes only 
Score using the following scale: 

7: guided use, using it a lot 

4: appropriate, right on target 

1 : no mention 

16. Which basic principles from the Metacourse were referred to: 

a. the paper computer 

b. purpose, syntax, action 

c. interaction between the user and the computer 

d. the minimanual 

e. patterns 

f. program production using metacourse principles 

g. other: 
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17. Percentage of the important r..aterial in this lesson 

that was adequately covered. 



Fidelity of lesson observed to lesson as written - 
7: taught almost exactly as written 

4: the same or very similar content of lesson as written, but adapted 
and paraphrased 

1: almost unrecognizable as the same lesson 



18. Introduction 

19. Lecture 

20. Participation 

21. Explanation 

22. Exercise 
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Appendix G 
Student Questionaire 
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Previous Coniputinq Experience 



NAME 



Please answer the following questions concerning your experience with^ 
computers. If you have a hard time answering any of these questions just 
give it your best guess. The information will help us in our research, 
and we appreciate your help with this project. 



1) Have you ever taken a BASIC programming course before this semester? 
Yes No 

2) If you answered yes to number 1: When did you take the course(es), how 
long did the course(s) last and how often did the class(es) meet? (For 
example, you might have had a BASIC course at a camp this past summer that 
lasted for 6 weeks and met 3 times a week for 2 hours each class.) 



3) Have you ever taken another programming course in a language other than 
BASIC? 

Yes No 

4) If you answered yes to question 2: What language(s) have you studied? 

Logo If yes, how much? 

Pascal If yes, how much?^ 

Other (name of language) If yes. how much? — 

5) In what grade were you first introduced to computer languages (Logo, 
Basic, etc)? 

6) In what grade were you first introduced to computer appl i cat ions (word- 
processing, data bases, spreadsheets)? 

7) Is there a computer in your home? If so, 1) what kind is it, 2) how often 
do you use it, and 3) what 2 things do you use it for the most (for example, 
games, programming, wordprocessing)? 
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